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Survey Research in Educational Administration: A Critical Analysis 

Cecil Miskel and Terry Sandlin 
The University of Kansas 

An examination of the research literature in educational administra- 
tion, reveals a pervasive use of survey methods. A large portion of the 
published research includes a survey procedure. Given this pervasiveness, 
two important questions that need to be addressed are the following: 
What level of methodological rigor is evidenced by these studies? Did 
the rigor increase during the 1970s? The purpose of this paper is to 
assess the methodological merit of the published research in educational 
administration that has used survey procedures. 

To answer the questions, data were taken from a random sample of 24 
survey studies published in the Journal of Educational Administration 
and 23 in the Educational Administration Quarterly during the eight 
years of 1972-79. Each article was evaluated against six criteria for 
sampling and instrumentation procedures. Raw scores, descriptive statis- 
tics, and graphical analysis were used to assess the levels of quality 
and to test for systematic improvements in the scientific rigor of the 
published research. The deficiencies and strengths are presented and 
discussed. Qualitative differences between the journals and systematic 
trends were not evident. Finally, suggestions for future research are 
presented. 
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SURVEY RESEARCH IN EDUCATIONAL ADMINISTRATION: A CRITICAL ANALYSIS 
Several critiques have been made of the literature in educational 
administration. In particular, five analyses standout. After making a 
comprehensive review of the Educational Administration Quarterly ( EAQ ) , 
Campbell concluded that the announced purpose of the EA£ to publish con- 
ceptual, empirical, and analytic manuscripts has been largely achieved. 1 
Moreover, most of the articles published in the EA^ are of good quality, 
some of them of superior quality and certainly as well done as many of 
those in other journals. In his judgment, the more recent articles 
seemed to be of better quality. Campbell also noted that the Journal of 
Educational Administration ( JEA ) represents the only journal that truly 
competes with the EA£ for publishing manuscripts focused primarily on 
the field of educational administration. 

In a recent review of the research in educational administration, 

? 3 
Boyan agreed with Immegart and Boyd. They believe that to an increas- 
ing extent internal specialization characterizes inquiry in the field. 
In other words, scholars tend to specialize in theories guiding inquiry 
and methods of doing research. Clearly, survey methods represent one 
area of specialization in educational administration research. A cursory 
examination of the published research in the educational administration 
literature yields the observation that survey procedures constitute the 
most frequently used empirical method. In contrast to CampbeTs conclu- 
sion, Boyan believes that improvements in the states of inquiry in the 
field have been uneven and modest. 

In a more directed critique, McNamara analyzed the statistical 
methodology employed in the articles published in the first cweV j 
volumes of the EAQ. 4 McNamara concluded that univariate analysis tech- 
niques that test for differences between groups dominate the treatment 



of 'data. Moreover, tests were lacking to estimate the significance of 
the variable relationships; that is, post hoc procedures to determine 
the proportion of variance explained in the dependent variable v-are not 
in evidence. 

Apparently, some disagreement exists among the critiques regarding 
whether the scientific merit of the published research in educational 
administration is improving. Willson's evaluation of the research tech- 
niques used in the studies published in the American Educational Research 
Journal showed that over a ten year period (1969-1973) no broadening of 

the pool of research techniques had occurred, and deficiencies in random- 

5 

ization and unit of analysis remained. 

Given the pervasiveness of survey research, the limitations cited 
in earlier critiques and the possibility that the level of merit has 
^hanged, two questions that become important are: 1) What level of 
methodological rigor is evidenced- by the published investigations that 
have used survey procedures? 2) Did the methodological rigor increase 
during the 1970s? The purpose of this paper is to respond to these ques- 
tions with findings from a systematic evaluation of the articles report- 
ing findings of survey research studies that are found in the Educational 
Administration Quarterly and the Journal erf Educational Administration . 

Survey Research 

Survey research is a planned collection of data that consists of 
procedures used by investigators to enter a subject population and to 
measure a specific set of responses. As a branch of social scientific 
research, investigations comprised of sample survey methods examine 
large and small populations by selecting and studying samples chosen 
from the populations. 6 Although the purposes guiding specific studies 
vary, survey research usually provides descriptions, explanations, and 



predictions of relationships among sociological and psychological vari- 
ables—facts, opinions, attitudes, and behavior. 

To gather data, sample surveys employ questionnaires and inter- 
views, attitude scales, projective techniques, existent records such as 

7 

census data, and various related methods. Typically, independent vari- 

8 

ables are neither manipulated nor are control conditions employed.* 
Unfortunately, survey studies frequently fail to meet the common scien- 
tific criteria of quality and are too often conducted with insufficient 
planning. 9 The design and execution of a survey project not only re- 
quires technical expertise, but also arduous intellectual activities. 
Two areas of the design—sampling and measurement— are particularly suscep- 
tible to error and poor execution. 

Criteria of Quality 

Sampl ing 

Population . A population for any investigation is the total number 
of units in which the researcher is interested. A sample is a subset of 
the population which is drawn because it is impossible or impractical to 
work with all of the units in the intended population. 10 By definition 
survey research links populations and samples. Survey researchers study 
samples drawn from populations and generalize the characteristics to the 
specified population. 

The goal of sampling, therefore, is to select a smaller represent- 
ative subset of elements from the entire population. Consequently, de- 
fining the population is an essential step before sampling procedures 
can be formulated. Sudman suggests the use of the following character- 
istics when defining populations: geography, personal variables of age, 
race, education, institutional affiliations and intentions, and organiza-^ 
tional variables^ size, school level, and private or public type. 11 



However, the general izabil ity of the findings can be jeopardized by 

several components of the sampling design including the unit of analysis, 

participation rate, and method of sampling. 

Unit of analysis . According to Kish, the unit of analysis is the 

fundamental element of the population for which information is being 
12 

sought. Perhaps a more descriptive designation is the unit of invest- 
igation. In other words, unit of analysis refers to the fundamental 
elements of the population about which inferences are to be drawn. 

As noted by Burstein, the hierarchical nature of educational organ- 

13 

izations produces several levels that can serve as focal units. For 
example, appropriate units of analysis for research in educational admin- 
istration include individuals, classrooms, curriculum programs, school 
attendance centers, and districts. Moreover, the selection of the appro- 
priate unit(s) of analysis should be based on the theoretical formula- 
tion of the study. 14 If the variable "o be studied are concerned with 
administrative organization and processes in school buildings, for in- 
stance, the attendance center is a more appropriate unit of analysis 

1 5 

t^an the individual . 

The unit of analysis or investigation directly affects sampling 
procedures by defining the appropriate sampling element and the number 
of subjects in the study. If, for example, elementary students are the 
focus of the study and hence the unit of analysis, a larger number of 
units exist and probably can be drawn more easily and in larger numbers 
than if the unit of analysis is the elementary school . Consequently, 
the specification by researchers of the unit of analysis allows a more 
accurate assessment of the adequacy of the sampling design. 

Participation rates . The survey population usually differs some- 
what from the target population. The primary difference 'frequently arises 



from nonresponses. Kerlinger, for example, concludes that the low 
return rates of mail questionnaires yield many studies that are worse 
than useless, except in highly sophisticated hands- 17 The problem is 
v not in just the number of those participating, but the problem of poten- 

tial bias in those returning questionnaires. When mail questionnaires 

are used, a general agreement seems to exist that the return rate should 
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approach 80 percent or higher. Even with such a high return rate, the 

19 

representativeness of the responses may remain questionable: 

Methods of sampling , A major source of sampling error occurs be- 
cause of the sampling design itself. Potential sampling designs include 
convenience, systematic, simple random, and probability selection tech- 
niques. In many cases s combinations of the procedures are employed. 

Convenience'sampling is selecting a particular subgroup within the 
population. because that subgroup is easily accessible. This technique 

contains the assumption that the available respondents are representa- 

20 

tive of the total population, which sometimes is not true. Neverthe- 
less, convenience samples often are necessary and unavoidable. Kerlinger 
notes that the weaknesses of convenience samples can somewhat be reduced 
by the use of knowledge and care in selecting the samples, along with 
replicating studies with different samples.^ 

In systematic sampling, the first sample element is randomly chosen 
from a list, and subsequent units are chosen at regular intervals from 
the list. Two basic assumptions undergird this procedure: the list is 
arranged randomly, and, the feature by which it is arranged is not related 
to the purpose of the survey. If these two assumptions are met, most 
systematic samples exhibit the same precision as random samples. System- 
atic samples sometimes are designated H pseudo-simple random samples" or 

22 

"quasi-random samples. 11 
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Random sampling occurs when each element in a population has an 
equal chance of being drawn, and all possible samples have an equal 
chance of being drawn. The advantage of using random samples is that 
constant and independent probabilities are ensured. Therefore, random 
sampling forms the basis of adequate sampling procedures. Yet, in many 
cases, simple random sampling may not be sufficient to minimize error 
and, and more elaborate designs are required. 

Probability samples use some form of random sampling in one or more 
of their stages. The most common form of probability sampling is strati- 
fied sampling. In stratified sampling the population is divided into 
strata such as administrators and teachers, females and males, or elemen- 
tary and secondary schools. Random samples are then drawn from each 
stratum. Stratified sampling procedures are appropriate when specific 
strata are of interest or when prior information suggests differences 
among the strata. The purpose of stratified ^sampling is to reduce sam- 
pling error and to insure that the focal strata comprising the popula- 

24 

tion are represented in the sample. 
Measurement 

Oppenheim observed that, in general, great strides have been made 

in the improvement of sampling methods, but similar gains are not appar- 

25 

ent in questionnaire and interview development. Although many con- 
cerns exist about measurement techniques, Kerlinger agrees with Pfeiffer 
and Heslin who maintain that the most critical considerations are validity 
26 

and reliability. 

Validity . The basic questions of validity center on what charact- 
eristics are being measured, what the scores mean, and how useful the 
data are. As complex as the subject is, the most common definition of 

validity is epitomized by the question: Are we measuring what we think 
27 

we are measuring? 



In educational research, the three most important types of validity 
are content, criterion related, and construct. Content validity refers 
to the initial impressions which the user or judge has of the instrument. 
More specifically, content validity is the judged representativeness of 
the items; that is, the items adequately sample the content tapped by 
the measuring device. 

Criterion related validity is established by comparing test scores 
with one or more external variables, or criteria, that measure the--attri- 
bute under study. Predictive, concurrent, convergent, and discriminant 
validity are common types of criterion validity. 



Kerlinger asserts, however, 



that scientifically speaking, construct 
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validity is the most important type. Its significance resides in the 

l 

fact that construct validity unites psychometric techniques with theoretic 

/' 

concepts. Three parts comprise construct validity: indicating what 
constructs potentially explain test variance, deriving hypotheses from 
the theory involving the construct, and testing the hypotheses empiri- 
cally. 30 

Reliability . The basic questions of reliability pivot on concepts 

such as stability, dependability,' accuracy, and unsystematic or error 

variance. The basic definition of reliability is illustrated by the 

31 

question: How precise or accurate is the measuring instrument? In 

other words, the greater the consistancy of responses to items measuring 

the same concept, the greater the reliability. The more reliable the 

measure, the less random error it generates. 

Technically, the coefficient of reliability is the variance ratio 

32 

of true scores to total scores on equivalent forms of a measure. A 
number of different testing and statistical procedures have been pro- 
posed to provide coefficients of reliability. Stanley lists the follow- 
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ing as three major procedures. (1) Administration of two parallel 
forms and correlating the resulting scores. (2) Administration of the 
same measure at a later time and correlating the resulting scores. This 
is sometimes referred to as an estimate of stability test-retest relia- 
bility. (3) Subdivision of a single measure into two presumably parallel 
groups of items and correlating the resulting two scores. Spearman-Brown's 
prophecy formula and Cronbach's alpha represent examples of this approach. 
Regardless of how the calculation is made, the reliability coefficient 
is only an estimate of the percentage of total variance that can be de- 
scribed as actual variance and not due to error. 

In summary, four criteria of quality for sampling procedures and 
two for measurement techniques have been derived, defined, and discussed. 
These criteria were used systematically to evaluate a random sample of 
survey research studies. 

Methods 

Population and Sampling 

The population for the study was the articles using a survey re- 
search procedure that had been published' in 24 issues and eight volumes 
(8-15) of the Educational Administration Quarterly (EAQJ and in 16 issues 
and eight volumes (10-17) of t.hp Journal of Educational Administration. 
(JEA). These volumes were published from 1972 through 1979. The multi- 
level units of analysis were -he issue and the volume comprising the 
journal s. 

During the eight year period, 141 and 161 articles were published 
in the EAQ. and JEA respectively (see Table 1). Using the definition of 
survey research presented earlier in this paper, the articles were clas- 
sified as either employing a survey research procedure or being of an- 
other type. As an estimate of reliability, both investigators had to 

IF* 

-8- n 



agree that an article contained a survey component. As shown in Table 
1, both journals published a similar number ( EAQ = 64 and JEA = 66) of 
survey studies. To test whether the frequency of survey investigations 
per volume exhibited systematic trends , a chi square test of homogeneity 
was calculated. The chi square value was 4.67 (p>.05). Therefore, the 
number of survey articles per volume appears to be the same. 

Table 1 about here 

To ensure a representative sample, a stratified random sampling 
procedure was used. The population of survey articles was stratified by 
journal, volume, and issue. In the case of the EA£, one article was 
randomly selected from each of the 24 issues or three per volume. The 
exception was volume 10, issue 3, 1974 because it did not contain a sur- 
vey research article. Therefore, the sample from the EA([ included 23 
articles in 23 issues and eight volumes. 

Since a volume of the JEA is comprised of only two issues, one article 
was randomly selected from each issue and then a third article was chosen 
randomly from each volume. Thus, the sample from the JEA included 24 
articles in sixtean issues and eight volumes. 
Instrumentation! 

To operational ize the evaluation criteria of survey research methods, 
the nine item instrument shown in Table 2 was developed. As reviewed 
earlier in this, paper, four sampling and two measurement criteria served 
as a conceptual guide in building the instrument. The sampling criteria 
items measured the specification of the population, the specification of 
the unit of analysis, participation rates, and the type of sampling method. 
The instrumentation criteria scale items assessed the specifications of 
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validity and reliability estimates. Similar to the measurement proce- 
dure used by Wi 1 1 son, instruments not subject to reliability analysis in 
the usual sense of psychometrics were not considered. The most common 
examples of this type of measure is the use of published figures and 
demographic information such as sex, age, and job assignment. Three 
general items (5,8,9) were created by combining and averaging the values 
of two or more items. 

Table 2 about here 

To quantify the quality of each criterion three categories were 
developed for each item that indicated poor, adequate, and good survey 
methods. The three categories were assigned scale values of 0, 1, and 2 
respectively. The definitions and category values for the items and the 
two scales are presented in Table 2. In addition, the four items com- 
prising the sampling criteria scale were averaged to yield item 5, an 
indicator of overall adequacy of the sampling procedures. The possible 
range of scores was 0-2. Similarly, the two item scores for the instru- 
mentation criteria were averaged to produce item 8, a summary statistic 
for measurement adequacy. The possible range of scores was 0-2. Finally, 
an overall assessment of the survey methodology was calculated by averag- 
ing item 5 (sampling criteria scale) and item 8 (measurement criteria 
scale). The range, therefore, was 0-2 with high scores indicating the 
use of better survey methods than low scores. When a study used more 
than one population, sample, or measure, the values were averaged to 
produce a single score for each item. The scores for items for the in- 
dividual articles in each issue and volume were aggregated *o create 
scores for the 23 issues and eight volumes of the EA£ and the 16 issues 
and eight volumes of the JEA. 

-10- 
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Application of the measure given in Table 2 was accomplished by the 
two investigators independently rating each article on the eight items. 
I itercoder reliability was attained by calculating the percentage of 
agreement for each item. The average percentage of intercoder agreement 
across the eight items was 96% and the range was 95% to 98%. Intracoder 
reliability was estimated when each investigator receded the data one 
week later. Intracoder agreements over all the variables were 97% and 
98%. 

A panel procedure was used to establish content validity for the 
instrument. Three research methodol ogi sts agreed that the six individual 
items represent important criteria for evaluating survey methods. 
Analysis 

To answer question one regarding the quality of research, raw scores 
and descriptive statistics were used. Specifically, means and standard 
deviations were calculated separately for each indicator of quality across 
the issues and volumes of both journals. To respond to question two 
about the possibility of a trend in the level of methodological ri ]or 
during the 1972-79 time period, the scores for the sam^ing criter.a and 
measurement criteria. Moreover, these procedures were applied at two 
levels or units of analysis, that is, by issue and by, volume for each 
journal . 

Findings 

The two questions guiding the research are addressed separately. 
In addition, the findings are presented for the two levels of analysis 
and for each journal. Comparisons across levels and journals complete 
the response to each question. 
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Question One: What level of methodologial rigor 
is evidenced by the published investigations 
that have used survey procedures? 

Issue level of analysis . Table 3 presents a summary of data for 
the 23 issues of the EAQ_. The scores for each criterion across the issues 
comprise the nine numbered columns, while the scores for each issue across 
the nine criteria form the rows. Means and standard deviations for each 
criterion across the issues are given at the bottom of Table 3. 

Table 3 about here 

Based on the values of means for the EAQ_ issues on the sampling 
criteria, only two of the four indicators had an average value that can 
be described as adequate to good (see Table 3). The population criterion 
(column 1) had a mean of 1.33. The population was specified in six of 
the 23 issues, and in all cases the samples were discussed adequately. 
Similarly, the methods of sampling (column 4) tended to be adequate with 
a mean of 1.36. Either a random sampling procedure or the entire popula- 
tion was typically used. However, in six articles it was impossible to 
determine the selection techniques. In contrast, the specification of 
the unit of analysis (column 2) was poorly detailed and had a mean of 
.20 on a .00-2.00 scale. The unit of analysis was mentioned in only 
five investigations, and multilevel analyses were conducted in only one 
study. Moreover, the unit of analysis in a dominate portion of the issues 
was the individual, student, teacher, or administrator. The 23 studies 
also demonstrated less than adequate quality on the criterion for the 
participation rates (column 3) with a mean of .75. Eight issues either 
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failed to provide adequate information to calculate the return rates or 
had less than a 60% level of participation. As indicated by the mean of 
.91 for the sampling criteria scale (column 5), the overall quality of 
the sampling procedures in the EA£ has been marginally adequate. 

The quality of measurement in the EA£ studies varies with the cri- 
terion. Reliability (column 6) wit,", a mean of 1.05 is addressed more 
systematically than validity (column 7) with a mean of .57. Moreover, 
many of the articles refer to reliability and validity by citing other 
studies or mentioning in a vague fashion that reliability and validity 
had been established. As indicated by the mean of .81 for the measure- 
ment criteria scale (column 8), the overall quality of the measurement 
procedures in the EA£ has been less than adequate. 

Table 4 displ ' . summary of the data for the 16 issues of the 
JEA. Only two of the four sampling criteria can be considered adequate. 
Specification of the population has a mean of 1.19, and the sampling 
method criterion attained a mean cf 1.30. The lowest possible score of 
zero was calculated for the unit of analysis criterion. No study explic- 
itly stated what the focal unit was, and none used multilevel analyses. 
Similarly, the rates of participation were poor with a mean of .55. 
Overall, the sampling criteria scale mean of .71 indicates that the sam- 
pling procedures exhibited by the articles published in JEA tended to be 
inadequate. 

The two means for the quality of measurement criteria in the JEA 
investigations attain similar levels. The reliability criterion with a 
mean of 1.07 reached adequacy, while the validity criterion approached 
the adequate level with a mean of .85. With one exception for relia- 
bility and two for validity, each issue addressed these criteria to some 
extent. As Indicated by the mean of .96 for the measurement criteria 



seals, the quality of the measurement procedures for the JEA was margin- 
ally adequate. 

At the issue level of analysis, the EAQ. and JEA exhibit similar 
strengths and weaknesses on the survey research criteria (see Tables 3 
and 4). Both publish articles that show the highest quality on specify- 
ing the population, using random sampling procedures, and providing esti- 
mates of reliability. Conversely, both publish articles that are inade- 
quate in specifying the unit(s) of analysis and giving or achieving suf- 
ficient participation rates. The JEA articles tend to explicate the 
validity of measures more adequately than those in the EAQ.. Slightly 
better scores on the sampling criteria scale were attained for the EAQ 
(X = .91) than for the JEA (X = .71). On the measurement criteria score 
the reverse relationship holds with the JEA having a mean of .96, and 
the EAQ. having a mean of .81. The overall criteria scale scores are 
essentially equal (EAQ, = .86; JEA = .84). 

Volume level of analysis . Tables 3 and 4 summarize the data by 
volumes for the EAQ. and JEA respectively. Aggregating the issue level 
data to the volume level produced few variations in the results. The 
largest change for the EAQ. was a decline in the mean from 1.05 to .93 
for the reliability of measurement criterion. The largest change in the 
JEA was a decline in the mean from 1.30 to 1.03 for the sampling method 
criterion. All other changes were minimal. 

Summary of findings for question one . The levels of methodological 
rigor are not high. With a score of 1.00 being defined as adequate, 
neither the EAQ. nor JEA attained this level on the sampling criteria 
scale, measurement criteria scale, or overall quality scale. Therefore, 
the survey research rigor approaches, but does not attain adequacy in 
the two journals. 

-14- 
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Question Two: Did the methodological rigor 
Increase during the 1970s? 

Issue level of analysis . Graph 1 pictorially displays the means of 
sampling and measurement criteria scales across the 23 issues of the 
EAQ . No trends are apparent in either scale. The data points bounce 
widely and show no tendancies to form a narrower band or to become higher 
over tdme • 

Graph 1 about here 

\ 

Graph 2 presents the means of the sampling and measurement criteria 
scales across the 16 issues of the JEA. No trends are evidenced for 
either scale. 

Graph 2 about here 

Volume level of Analysis . Craph 3 exhibits the data summaries for 
the sampling and measurement criteria scores for the eight volumes of 
EAQ . While no trends exist for improved rigor, volumes 11 through 15 
show more stability than volumes 8 through 11. For the later volumes, 
the means show less variability but remain at or somewhat below 1.0. 
However, the means on the measurement criteria sc : declined for volumes 
14 and 15. 

\ 

i 

/ 

t 

Graph 3 about here 

< i 

Graph 4 plots the sampling and measurement criteria means across 

the eight volumes of JEA . No trends are evident for criteria, and the 

variability does not seem to lessen in the later volumes. 
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Graph 4 about here 



Summary of the findings for question two . The results from graphing 
the data show no increased or decreased rigor during the 1970s in the 
quality of survey research methodology in the EAQ or JEA. In fact, the 
data plots for both journals suggest wide variations in quality from 
issue to issue and from volume to volume. 

Discussion 

— ■ — \ 

Level of Methodological Rigor 

Even a cursory examination of data discussed earlier and presented 
in Tables 3-6 reveals several important characteristics about the quality 
of the survey research published in the EAQ and JEA . Perhaps most alarm- 
ing finding is the lack of consideration given to the unit of analysis. 
Only five EAQ articles and no JEA articles explicitly mentioned the focal 
unit of the study. With one exception, the investigations used only a 
single unit of analysis. Typically, the focus was on the individual 
with no evidence of the data being aggregated to the classroom, school 
building, or district. *This result; is similar to the finding of Willson 

that few studies published in the American Educational Research Journal 

- 35 \ 

recognize different aggregation levels. Potentially, a large amount 
of information from different perspectives is being lost by the practice 
of using a single unit of analysis. Greater emphasis should be placed 
on organizational levels such as the school attendance center and dis- 
trict in research for the field of educational administration. 

The reporting of research with either low participation rates or no 
mention of the rates occurs too frequently. In particular, over one-half 
..of the JEA issues and over one-third of the EAQ issues failed to meet 
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minimal standards on this criterion. Inadequate participation rates 
certainly calls into question the generalizability of many of the find- 
ings that appear in the educational administration literature. 

Hardly any of the investigations specifically defined the popula- 
tion of interest. Rather, emphasis was typically placed on describing 
the sample. After reviewing a large number of studies, a somewhat cyni- 
cal impression emerges that the elaborate, description of the sample is 
part of an effort to suggest that the findings are highly generalizable. 
Yet in irany cases the population is nisc mentioned, and the participation 
rate is low or not ide..itifie.I. "J'h'.s condition is poor practice even 
when the intent icus are positive, 'out it is an unethical practice when 
the intentions are negative. 

As cited earlier, Oppenheim has observed that great strides have 

36 

been made in the improvement of sampling methods. The scores on the 
sampling criteria items and scale, however, suggest that the improve- 
ments have not bien applied systematically in the field of educational 
administration. Either many researchers have not learned of the ad- 
vances, or .hey have chosen to ignore them. 

Inadequacies also exist in the quality of the survey measures. 
While references to the reliability estimates resulted in relatively 
large *ean scores of about 1.0, the finding must be tempered. Almost 
30% of the EAQ issues made no mention of reliability. A possible expla- 
nation for this omission is that investigators attached little impor- 
tance to reporting the reliability of previously developed and used mea- 
sures such as the OCDQ, LBDQ, PCII and so forth. On the other side of 
the issue, almost 40% of the EA£ meticulously provided empirical values 
for all of the instruments included in the study. In a sense, the re- 
porting of reliability estimates is either a feast or famine. 
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Mentioning and describing validity was particularly weak in the 
EAQ . About 40% failed to acknowledge this measurement criterion. How- 
ever, all but two of the JEA issues at least mentioned validity. 

An interesting difference between the EAQ and JEA should be noted. 
The overall sampling criteria values were higher for the EAQ , while the 
overall measurement values were higher for the JEA . In fact, the largest 
discrepancies between the EA£ and JEA occurred between the sampling cri- 
terion for participation rates and the measurement criterion of validity. 
EAQ had the highest participation rate scores, while JEA had the highest 
validity scores. Evidently the editorial boards emphasize different 
specific criteria. 
Trends in Methodological Rigor 

The findings of the current study regarding survey research pro- 
cedures do not support Campbell's conclusion that recent articles in the 

EA^ are improving. 37 Rather, the results reported in this paper suggest 

38 

th?- Boyan's ssertion reflects the state of research. Improvements 
have been uneven and modest. Perhaps the topics, problem definitions, 
and use of conceptual models are better in more recent volumes of the 
EAQ and JEA . But the sampling and measurement criteria of quality show 
wide fluctuations and demonstrate no discernable trends. Consequently, 
the present findings support the assertion that the levels of quality 
remain eratic for the survey research procedures used in the field of 
educational administration, , 
Related Observations 

Although the concept of specialization of methods as mentioned by 
Imroegart, Boyd and Immegart, and Boyan was not addressed directly, sev- 
eral names. seemed to appear more frequently than would normally be ex- 
pected. 39 This observation provides some support for the idea that some 
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researchers in educational administration specialize in using survey 
research methods. Since over 40% of the published articles in the EA£ 
and JEA are survey based, the contention could be made that the field 
itself specializes in survey research. 

Comparing the scores of the two journals on survey research criteria 
reveals few substantial differences in quality. The EAg seems to empha- 
size sampling adequacy, while the JEA tends to insist on at least men- 
tioning validity in each study. Both suffer general shortcomings across 
a number of the quality criteria. 

Conclusion 

Two common approaches to the evaluation of research are to find 
fatal flaws or redeeming features. In a sense the present study has' 
focused on fatal flaws. The blame for the observed weaknesses of the 
EAQ and JEA should not be placed on the respective editors and editorial 
boards, however. The two journals probably reflect the general sophis- 
tication level of the field and publish the best 'survey research articles 
that are available in educational administration. Many of the invest- 
igations had several redeeming characteristics. Indeed, the quality 1 
ranged from excellent to abysmally low. 

The editors and editorial boards can improve the quality of presen- 
tation if not the quality of the methodology itself. ■ They should demand 
that the researchers provide a clear exposition of the methods that were 
used. There is no reason for the methodology to be ^eabed s o tersely 
that the readers can not evaluate the quality of procedures! 
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Finally, the quality criteria for survey research are well devel- 
oped and sources are available to guide the design of research projects. 
For most studies, slight increases in technical expertise, planning, and 
physical effort would improve the survey research procedures immensely. 
Scholars in educational administration should make the commitment to 
improve the methods and hence the quality of our knowledge base. 
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Table 1 

Frequency and Percent of Survey Articles 
Published in Each Journal by Year 



Educational Administration Quarterly Journal of Educational Administration 





Total 


Survey 


Percent 


Total 


Survey 


Percent 


Year 


n.L L1L> lcb 




r>f Total 


Ar t icles 


Based 


of Total 


1972 


16 


8 


50.0 


15 


6 


40.0 


1973 


16 


10 


62.5 


24 


7 


29.2 


1974 


18 


6 


33.3 


20 


10 


50.0 


1975 


18 


6 


33.3 


21 


10 


47.6 


1976 


18 


11 


61.1 


22 


7 


31.8 


1977 


18 


9 


50.0 


21 


9 


42.8 


1978 


19 


9 


47.4 


19 


6 


31.6 


1979 


18 


5 


27.8 J 


19 


11 


57.9 


Summary 


141 


64 


45.4 


161 


66 


41.0 
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Table 2 

Scales and Items Comprising the Survey Research Evaluation Instrument 



SAMPLING CRITERIA SCALE 

1. Specification of the Population 

___ 0. Poor. The population was neither identified nor was the 
" sample described, 

1. Adequate, The population was not mentioned, but it could b 
~ inferred from the description of the sample. 

2. Good, The population was explicitly identified. 



2. Unit of Analysis 

0. Poor. The unit of analysis was not mentioned, but an 

identifiable single level unit was used. 

1. Adequate. A single level unit of analysis was identified, 

~ or multilevel units of analysis were used. 

Good. Multilevel units of analysis were explicitly 

identified and used. 

3. Participation Rates 

0. Poor. The participation rate was not specified or less 
han 60%. 

1. Adequate. The participation rate was between 60-80%. 

2. Good. More than 80% of the sample participated. 



4. Sampling Method 

0. Poor. The method of sampling was either not specified 
" ~~ or convenience procedures were used. 

1. Adequate. A mixture of convenience and random selection 
~ procedures were used. 

2. Good. Random selection procedures were used. 

5. Sampling Criteria Scale (Sum of items 1-4). 

Mean of Items 1-4. 
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Table 2 (continued) 
Scales and Items Comprising the Survey Research Evaluation Instrument 



MEASUREMENT CRITERIA SCALE 

6. Reliability 

0. Poor. Reliability was not mentioned. 

1. Adequate. Reference was made to standardized instruments, t 
previous studies or to the type(s) without specifying values 

____ 2. Good. Empirical values were reported. 



7. Validity 

0. Poor. Validity indicators were not mentioned. 

1. Adequate. Reference was made to standardized instruments, to 
previous studies, or to the type(s) without describing the 
procedures • 

2. Good. The type(s) and procedures used to establish validity 
were described. 

3. Measurement Criteria Scale 
Mean of Items 6-7. 



9. Overall Quality 

Mean of items 5 and 8. 
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Table 3 

Summary of the Data for the Evaluation Criteria for the Survey Studies 
Published in 23 Issues of the Educational Administration Quarterly 



Sampling Measurement Overall 



Issue Volume 


1 


2 


3 


A 


5 


6 


7 


8 


9 


1 


8 


2.00 


.00 


1.00 


2.00 


1.25 


.63 


.38 


.51 


.88 


2 


8 


1.00 


1.00 


1.33 


2.00 


1.33 


.00 


.00 


.00 


.67 


3 


8 


2.00 


.00 


1.00 


2.00 


1.25 


.00 


.00 


.00 


.63 


H 


q 


1.00 


.00 


.00 


.0.0 


.25 


1.00 


1.00 


1.00 


.63 


5 


9 


1.00 


c00 


1.00 


1.00 


.75 


2.00 


1.00 


1.50 


1.13 


6 


9 


,1.00 


.00 


.00 


.00 


.25 


2 .00 


2.00 


2.UU 


-l 1*3 

1 • 1j 


7 


10 


a. so 


.00 


2.00 


2.00 


1.38 


.00 


.00 


.00 


.69 


8 


10 


2.00 


.00 


1.00 


2.00 


1.25 


.33 


.33 


.33 


.79 


9 


11 


1.00 


.00 


1.00 


2.00 


1.00 


2.00 


.63 


1.32 


1.16 


10 


11 


1.00 


.00 


1.00 


1.00 


.75 


.50 


.50 


.50 


.63 


11 


11 


2.00 


.00 


1.00 


2.00 


1.25 


2.00 


.00 


1.00 


1.13 


12 


12 


2.00 


1.00 


.00 


2.00 


1.25 


.00 


.00 


.00 


.63 


13 


12 


1.00 


.50 


.00 


1.33 


.70 


2.00 


1.00 


1.50 


1.10 


1A 


12 


1.00 


1.00 


1.50 


2.00 


1.38 


2.00 


1.00 


1.50 


l.AA 


15 


13 


1.00 


.00 


.00 


2.00 


.75 


2.00 


2.00 


2.00 


1.38 


16 


13 


2.00 


.00 


1.33 


2.00 


1.33 


1.67 


.83 


1.25 


1.29 


17 


13 


1.00 


.00 


.00 


.00 


.25 


.00 


.00 


.00 


.13 


18 


1A 


1.00 


.00 


.00 


2.00 


.75 


.00 


.00 


.00 


.38 


19 


1A 


2.00 


.00 


1.00 


.00 


.75 


2.00 


1.00 


1.50 


1.13 


20 


1A 


1.00 


.00 


.50 


.00 


.38 


1.00 


1.00 


1.00 


.75 


21 


15 


1.00 


.00 


.00 


.00 


,25 


1.08 


.33 


.71 


.A8 


22 


15 


1.00 


1.00 


.50 


2.00 


1.13 


2.00 


.00 


1.00 


1.07 


23 


15 


1.00 


.00 


2.00 


2.00 


1.25 


.00 


.00 


.00 


.63 




Mean 


1.33 


.20 


.75 


1.36 


.91 


1.05 


.57 


.81 


.86 


Standard 
















.69 


.3A 


Deviation 


.A7 


.39 


.66 


.88 


.Al 


.88 


.62 



Note ; 1 - Specification of the Population; 2 - Specification of the Unit 
of Analysis; 3 » Participation Rates; 4 = Sampling Method; 5 = Sampling 
Criteria Scale; 6 - Reliability; 7 - Validity; 8 - Measurement Criteria 
Scale; 9 - Overall Quality Scale. 




Table 4 

Summary of the Data for the Evaluation Criteria for the Survey Studies 
Published in 16 Issues of the Journal of Educational Administration 



Sampling Measurement Overall 



Issue Volume 


1 


2 


3 


4 


5 


6 


7 


8 


9 


1 


10 


1 00 


.00 


84 


2.00 


.96 


1.88 


1.50 


1.69 


1.33 


2 


10 


i nn 


. uu 




on 


25 


1 00 


2.00 


1.50 


.88 


3 


11 


1 nn 


nn 


.00 


1.00 


.50 


.50 


.50 


.50 


.51 


4 


11 


2.00 


.00 


.00 


2.00 


1.00 


.00 


.00 


.00 


.50 


5 


12 


1.00 


.00 


1.75 


2.00 


1.19 


.25 


.00 


.13 


.72 


6 


12 


1.00 


.00 


.00 


2.00 


.75 


1.00 


1.00 


1.00 


.88 


7 


13 


1.50 


.00 


.75 


1.00 


.81 


1.38 


.38 


.88 


.82 


8 


13 


1.00 


.00 


.00 


.00 


.25 


2.00 


2.00 


2.00 


1.13 


9 


14 


1.00 


.00 


.00 


1.00 


.50 


1.00 


1.09 


1.05 


1.78 


10 


14 


1.00 


.00 


.00 


.00 


.25 


2.00 


1.00 


1.50 


.88 


11 


15 


1.00 


.00 


2.00 


2.00 


1.25 


1.00 


1.00 


1.0 


1.13 


12 


15 


1.50 


.00 


1.50 


1.00 


1.00 


1.50 


.84 


1.17 


1.09 


13 


16 


1.00 


.00 


.00 


.50 


.63 


.25 


.25 


.25 


.44 


14 


16 


1.00 


.00 


.00 


.00 


.25 


.78 


.33 


.56 


.41 


15 


17 


2.00 


.00 


.00 


.00 


.50 


1.00 


1.00 


1.00 


.75 


16 


17 


1.00 


.00 


2.00 


2.00 


1.25 


1.50 


.75 


1.25 


1.19 




Mean 


1.19 


.00 


.55 


1.30 


.71 


1.07 


.85 


.96 ' 


.84 


Standard 




















Deviation 


.36 


.00 


.80 


.87 


.37 


.62 


.61 


.56 


.28 



Note ; 1 - Specification of the Population; 2 » Specification of the 
Unit of Analysis; 3 « Participation Ratea; 4 - Sampling Method; 
5 » Sampling Criteria Scale; 6 - Reliability; 7 - Validity; 8 - Measure- 
ment Criteria Scale; 9 ■ Overall Quality Scale. 



j Table 5 

Summary of the data for the Evaluation Criteria for the Survey Studies 
Published in the 8 Volumes of the Educational Administration Quarterly 



Sampling Measurement Overal l 



Volume 


Year 


1 


2 


3 


4 


5 


6 


7 


8 


9 


8 


1972 


1.67 


.33 


1.11 


2.00 


1.28 


.21 


.13 


.17 


.73 


9 


1973 


1.00 


.00 


.33 


.33 


. .42 


1.67 


1.33 


1.50 


.96 


10 


1974 


1.75 


.00 


1.50 


2.00 


1.32 


.17 


.17 


.17 


.74 


11 


1975 


1.33 


.00 


1.00 


1.67 


1.00 


.83 


.38 


.94 


.97 


12 


1976 


1.33 


.83 


.50 


1.78 


1.11 


1.33 


.67 


1.00 


1.06 


13 


1977 


1.33 


.00 


.44 


1.33 


.78 


1.22 


.94 


1.08 


.93 


14 


1978 


1.33 


.00 


.50 


.67 


.63 


1.00 


.67 


.83 


.75 


15 


1979 


1.00 


.33 


.83 


1.33 


.88 


1.03 


.11 


.57 


.73 


Mean 


1.34 


.19 


.78 


1.39 


.83 


.93 


.55 


.78 


.86 


Standard 




















Deviation 


.27 


.30 


.41 


.61 


.31 


.52 


.44 


.45 


.13 



Note : 1 « Specification of the Population; 2 » Specification of the 
Unit of Analysis; 3 - Participation Rates; 4 » Sampling Method; 
5 = Sampling Criteria Scale; 6 » Reliability; 7 « Validity; 8 =» Measure- 
ment Criteria Scale; 9 * Overall Quality Scale. 
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Summary of 



Published tu tfl 



t he ^ \ the EV*l uati °n Critef ia f or s urvey Studies 
a* Vrv. *he Jon>^ rational .... 



Administration 




Mean 

Standard 
Deviation 



/ Note- 1 ' "^Ic.' 1 "" ° f *Wf» 2 " ^Wlcatlon of 
the Unit o t An/* 1 *' 3 . Wrtl^K 4 * Method; 

„e„t Criteria O^" 11 Seal- 




2 4 6 8 10 12 14 16 18 20 22 

Issue 

Graph 1. Level of Quality on the Sampling and Measurement 
Criteria Scales for Twenty-three Issues of GAQ 
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O" " o Sampling Criteria Scale 

qmimimid Measurement Criteria Scale 
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o Sampling Cr' ter ia scale 

a Measurement c rit er ja Scale 
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Volume 

Graph 3. Level of Quality on the Sampling and Me asur erY ie nt 
Criteria Scales for Eight Volumes of EA^ 

1 
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1.8 " 
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1.4 



CO 
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o 
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o « o Sampling Criteria Scale 
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Volume 
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Grapgh 4. Level of Quality on the Sampling and Measursr >nt 
Criteria Scales for Eight Volumes of JEA 
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